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ABSTRACT • ^ • 

Concommitant with changes in ed 
roles r and evaluation in recent years ^ a new si 
and M.D. program was inititated at the TJniversi 
Kansas City. Evaluation was integrated into the 
order to assess student progress toward the ove 
a safe physician>„ as well as achievement in spe 
curriculum. Although several evaluation procedu 
this institution^ this paper describes only the 
Item Library is a computerized information retr 
containing, at this, timer lO-vOOO test questions 
information. Each entry of -the Item Library con 
guestion> Hedical Subject Heading (MeSH) coder 
sourcer history of use and upd^te'r item analysi 
bibliographic reference^ The psychometric data 
Quarterly Profile Examo generated from the Item 
and valid measures. Extending the computer assi 
procedures of the Item- Library into other allie 
progress. (Author/RC) 
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In recent years several, developments have occurred re- 
garding the education and role of members of the health care 
team. '^Developments in the educational process included com- 
bining programs in a manner which cuts across the traditional 
jiepartments and disciplines. The. process of integrating prd-^ 
grams yielded curricula which include cominunity involvement . • ' . 

more so than in the P^t^,- As the c;on9ept of the health care 
team has evolved, nev^ and expanded roles have de^veloped. For 
example, the role of the physician assistant has , developed 
within the past decade. Continuing education and career mo- 
bility have been emphasized recently much more sd than in the 
^ast. The December, 1975 Journal of Medical . Edudation is com- 
mended <to your reading for more information on the above points.. 5 

Concomitantly, related developments in education occurred. \ 

■ \ • 

There has been a movement to specify curriculum goals and ob- 
jectives in far greater detail than in past. Testing of I 

student mastery of information^has been geared to these cur- ! 
riculum objectives. " Furthermore, the computer has been intro- 
duced to curriculum, evaluai;j.on , and instructional aspects of 
the educational process. The prpgram described : this pre- • 
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sentation reflects many of the trends in health and education 
described thus far. 

.The University of Missouri - Kansas City School of Medl'cine 
admits students" 'directly from high school to a six-year com- 
bined undergraduate-medical education program. One unique fea- 
ture of the curriculiam organizatio^i is the assignment of students 
in srtiall groups to physician-instructo-rs called "docents". Each 
docenc has responsibility for the medical content ,of the curricu- 
lum. In' the .first two years ^of study ,^ liberal arts and medicine 
represent 7S% and 25% respectively of the curijiculum.^ During 
year^. ^yiree Lhrpugh six these proportions are reversed. Thus, 

. . - 

Students receive their "bachelors and med-ical degrees simultaneously 
at the conclusion of the progranr. 

, J .Evaluation was integrated into the academic plan at this 
nfew Schoo'L of Medicine to assess student progress toward the 
overall goal of becoming a 3afe physician, as well, as achieve- 
ment in specific areas ^of the curriculum. The evaluation pro- 
cess is frequently used to make prescriptive statements about 
individual student's curriculum plan. ^ This paper focuses on' 

one aspect of the overall evaluation process, the Item Library. 

... 1 

In the next several Mnutes, I will describe the Item Librar^y, 
discuss its functions!, propose its use in health care programs , 
describe psychometric properties of exams generated from the 
item Library, and/ discuss the significance of our use of com- 
puter technology. 

The Item Library is a computerized information retrieval . 
system containing ^000 test questions and support information; " 
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the number continues to increase each month. . Each item of the 
Item Library consists of the test question, Medical Subject 
Heading (MeSH) code, date submitted.^ source," history of use, 
item ""analysis data, and a specific bibliographic 'reference . 
Slj^_J^_llJr ti3LiaLBJ^ the conteuL ol each ifaem. Some of these" 
components warrant additiqnal comment. 

For our purposes,^ a "good'* test question should include 
a statement of fact, the question, and a set of five response 
alterna'ti '/es . Faculty, students, and visiting .docents contri- 
bute test questions relevant to their field of specialty. The 
quantity and quality of contributions has steadily increased 
with time. The School of Medicine at the University of Wiscon- 
sin and tha Rockford School .of Medicine have been generous in 
allowing us access to their question banks. The 7000 test 
questions currently in the Item Library represents less than 
the number of the questions on hand, because each question is 

extensively reviewed, referenced and caded for MeSH before en- 

■ ■ ■ , \, . 

tered in the computer. 

The MeSH code is a tree structure indexing method developed 

by the National Library of Medicine- Each test question is 

assigned at Teast one ,^nd no more than twelve Me.gH codes , ac- . 

cording to its content. Item analysis data is the proportion 

of examinees responding to each, answer choice at years. .1^6 and 

f acull-y/staf f for the most recent use of the questions and com- 

bined use of the questions. An index of item reliability w Lli ( 

- bo addod cqo h. (x($C 4v v^eJ^ • 
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• The item Library is us.ed to generate our Quarterly Pro- 
file Exam and to provide self-assos'sment learning experi,ences 
\;'ia remote terminals. The Quarterly Profile Examination (QPE) 
-is a. set of 400 questions intended to assess factAial knowledge 
in the area of medicine. Each quarter the QPE is generated ac- 
cording to specifications of content: area (MeSH codes) and 
the proportion of old/new questions. The MeSH codes appropriate 
for five major content areas. Internal Medicine, Pediatrics, 
Ob/Gyn^ Surgery, and Basic Science are used to select the test 
items. Usually, 80% of .the test ^questions have not been used 
in a previous QPE. Since t^st questions generally have more 

than one MeSK code, we are ■B'&le to establish additi.onal cate- 

.% 

gories f or - sco^r^jng. A -given test question may, therefore, con- 
tribute to the score of more than one category. The list of cate- 
gories and their appropriate MeSH code(s) is -shown in slide 2. 

The QPE is administered to all students, years one through 
six,' as well as interested faculty, staff, and allxed heal.th 
professionals. Students are encouraged to view the exam as an 
assessment/learning experience. Since students leave the exam - 
with their exam booklet, references to most questiond, and rc 
ceive scores afid answer key in a few days, the likelihood of 
-viewing the exam as a learning experience is increased. T'hey - 

are aware, howeve::, . that the Council on Evaluation„wei.ghs 

sheavily QPE performance in their promotional considerations. Two 
flagging criteria are applied to each quarters' scores to facili- 
tate Council review: (1) bottom 15% of peer group and; (2) score 
below mean of class one year level under, peer group, t. 



At this point I should add that test questions are flagged 
on each exam, as well as studefnt's scores. Any item flagged by., 
one of thesjs two criteria is reviewed: (1) year 6 per cent cor-, 
rect minus year 1 per cent, correct less than 20%; and, (2) year e 
per cent correct: less than 60%. In addition '^to the flagging by 

9' 

item analysis data, challenges to the examination are encour- 
aged by the Council >on Evaluation. 

The self-assessment opportunity via remote terminals is 
by na^ture a learning experience. Ten cathode ray tube computer 
terminals are available for the students' use to access the . 
Item Library. In effect, students have access to the pool of- - 
question's from which their future QPE ' s will come. To generate 
a quiz at the terminal, one specifies number of questions and 
the. MeSH code(s) appropriate to the content. The content area 
may be as general or specific as the MeSH tree structure allows. 

For example, a student might seek questions for the general 

., ■ - 

area of Hemic and Lympathic Disease (C9) or questions on a topic 
-ars-specxfic-a-s'^'ThrolnBocytbs^^ . 109 Up to twelve 

MeSH may be combined with Boolean logic to •g'enerate a' search. < 
Then 'the question and set- of response alternatives are presented 
on the terminal' screen. After the student ' s response, the Cor- 
rect answer is indicated. The student may also ask for item 
— ana-lysi-s-data'^-on each ques'tion. — r . 

The potential application of the Item Library, conceptually 
and concretely, to other programs of medicine and other ;fields 
of thei health sciences appears promising. In- fact, we began 
about two years ago building into the Item Library the capa- 



bility of generating examinations for the non-physician pri- . 
mary care, practioner'l c The Profile Examination for the Physi- 
cian Extender (PEPE) will be the counterpart of the QPE. The 
next' portion of this presenilation will describe the major fea- 
tures of the PEPE p.raject. ' 

Rather than gear our generation of PEPE items to a single 
.physician extender curriculum, we took an approach which wpu].d 
apply to. programs in general. Dr. . Ned Smull and Ms. O'ane Kerbe 
developed a curriculuin outline which' represents the likely dida 
tic material of physician extender programs. Including major 
categories and their subdivisions, there are abotit 1000 ele-, , 
ments in the curriculum outline* MeSH codes were assigned each 
element of the curriculum outline. The next step was to study 
the curricula of physician extender programs and record the 
number of class hours devoted to the teaching of topics of the 
curriculum outline. The curriculiam outline, MeSH codes, and ; 
hours of teaching by . different programs forms what we call the- 
curriculum matrix. The accortjpanying slide illustrates the ma- 
trix. 

The curriculum matrix is the core upon which curriculum 
'documentation, item generation,, exam generation, score re- 
porting, and curriculum prescription depend. The curri^ulum_ 

matrix is "computerized" so that^it interfaces with the Item 
Library. Currently we use this system to conduct periodic in- 
ventories of the Item Library by content. • This information 
^guides our item generation efforts by identifying content areas 
with' few items. The- system has also been used to generate pro- 



totype PEPE's. The MeSH codes Assigned to elements of the 
matrix were used to retrieve from the -Item Library exams re- 
presentative ot the curriculum content. 

How would anyone use this system to generate exams for 
their specific curriculiam?/' ..The answer is fairly straight- 
forward. Match your cur/iculum objectives with the PEPE cur- 
riculum matrix. Decide/total- lengthy of exam and proportional 
representation by coiitent area. Use MeSH codes and proportion • 
data to search the 2Ltem Library, 'obviously, the actual pro- > 
cess IS more diff/cult than the description leads one to 
believe; furthermore, we feel that a couple more years of ef- • 
•fort are necessary in order to develop the system's full ca- 
pabili1;ies Thus , the p'^resentation should be taken as a pro- 
gress rep^Drt on a project for which all of us concerned are 
excited, not a signal of completion. However, we have data to 
report on the QPE which will help you analyze the quality of 
the system. s . 

.Analysis of . the psychometric properties of the QPE is an. 
ongoing project. My next remarks summarize the analyses of 
reliability and validity. 

The reliability coefficients for the total score on all 



_.QPEl^-s.ince-^-May-, — ^73 ^have- been- 'ilf^tH^e range of . 95-. 98 when . 
computed by/the Kuder-Richardson formula 20. As. expected, the 
reliability of the various category scores is less than that- of 
the total score. The reliability of categories on the November 
'73 and March '74 exams is presented in slide 4, as repr^senta- 



"tfve^of usual results. Each quarter the Council on Evaluation 



ctinsiders the scores on Internal Medicine , Pediatrics , Ob/Gyn^ 
Sujgery, and Basic Science for the purpose of reviewing s;tudent 
progress. Of these 'five major categories, t)ne lowest observed 
reliability was .859. Recognizing the relationship between - 
"test length and reliability, the scores of minor exam categories 
are not considered for promotional* purposes . . - 

The preceding data indicates that the QPE has desirable 

■ - a , - ■ ■ . ' ■ ■ . 

properties of reliability. Now, let u's consider its validity. 
The' nature and function of the QPE is that of an achievement 
test; therefore, an appropriate . type of validity to consider is 
, content validity. The following points lead to a logical assump 
tion of adequate content validity: 

1, Curriculiam objectives were specified and coded for 

I* . _ ■ . 

- content. 

2. Questions of factual information were written and 
coded for content with the same MeSH code. 

^ 3. Each exam represents a sample of our pool of questions. 
While it is safe to assume content validity of the QPE in its 
present form,, two future refinements should improve content 
validity- of the exam: (1). we will attempt to have questions for 
every curriculunL_9^jejitive--pe^^feaii^^ information; 
and, (2) exam questions will be selected for various cpnte.nt • 
areas by a stratified random process. 

Mindful of the difficulty in assessing content validity wi 
statistical methods, I think data from our exam adds support to 
tii_e_as.SJamp±±on--of-^ori-^ First, the mean perfor- 

mance of groups on each exam is consistently of a pattern one 



would expect from an educational standpoint. In other words, 

mean performance has consistently. been dirpctly related to - 

, .- 

year level in school- Slide 5 illustrates this pattern for 
scores -on 'the August, .1973, exam. The second example is sCme- 
what.-mor4 involved. Half of the year two students take their 
6b/Gyn course Fall Semester ; the other year 2 'students, take 
Pediatrics. As a group the year two students taking the Ob/Gyn ^ 
course have scores significantly higher on the Ob/Gyn category 
than those students not enrolled in the course;, on the other 
hand, the Peds students have scored significantly higher in 
Pediatrics., The two groups did not differ significantly on 
any other category of the examination. Although it is impor- 
tant to* consider content validity of an^chievement test, cor- 
relations with external criteria also yield significant .infor- ' 
mation. A study of data f rom^ all^students (N=171.) conducted t&v^. 
Spring of 1973 indicated the QPE to correlate .43, . 36, and ^ 
.30 with clinical evaluations. Arts and Sciences GPA, and cur- 
"rent status in medical school, respectively. Multiple regres- 
sion analysis of 20 students' scores on National* Board of Medi- 
cine exam. Part 1, and/their QPE total scores on the two pre- 
vious exams yielded a multiple R of .827. 

Many of the previous remarks dealt with psychometric char- 
acteristics of our program for computerized* assessment of factual, 
information. Sound psychometric properties are -i^^ 
"for'^any^ evaluation procedure-. However, the practical usefulness ' 
of the assessment device must be significant enough to warrant 
its use, ho matter how great its psychometric properties. Also, >> 
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,when usj-ng* a computerized' system, a cost/benefit consideration 



becomes crucial,. At: this, point I would like, to comine]^ on the 
manner in which this testing program has be^n integrated into 
thQ educ'ational process/ Exara- scores are treated as indications 
of 'strengths and weaknesses.. Generally speaking a student's 
low 'scores are not seen: as low grades, but as. indications o£ ^ 
areas for future study. It is not uncommon for a gtudent's 
program to be ad-justed in .consideration of QPE performance, / " 

Another feature of the Item Library points to the inte- 
gration of testing into the educational process at UMKC, Cur- • 
riculum objectives, library reference materials, audio-visual, 
packages and the test questions are all coded for content by 
the MeSH code. Use of the MeSH code facilitates the integra- 
tion of curriculum, support material , and evaluation aspects of 
the program. This aspect of our procedure makes it. possible for** 
the Item Library to serve evaluation and learnij^g experience ' 
functions.. 
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(1) INFECTIOUS HOi«LEOSIS IS OFTEN A OIFF!Cl|LT DIAGNOSIS lO MAKE WITH 
ANY CEKTAINTY. IN ABOUT m OF CASES A' RASH MAY OCCUR WHICH. • 
SOHETIMES COMPLICATES .THE DIAGNOSIS FURTHER. THE RaSH USUALLY 

• ' RESEMBLES THAT 01-^: • 
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SU&kTl INTERNAL MEDICINE - 
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MEDICAL SUBJECT HEADING 

CATEGORY MeSH CODES 
Internal Medicine - G2 . 403 . 776 . 409 . 

Pediatrics G2 . 403 . 776 . 671 

Ob/Gyn G2 . 403 . 776 . 542 , G2 . 403 . 776 . 342 

Surgery G2 . 403 . 776 . 909 
Basic Science ; 

Physiology G1.782 

Biochemistry Gl . 201 

: ■ • 

Anatomy GJ..100 

Pharmacology G1.703 

Microbiology Gl. 273. 540 

Etc . 
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Computed by Kuder-Richardson 20 Formula 
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